

<!DOCTYPE html>
<html class="writer-html5" lang="zh" >
<head>
  <meta charset="utf-8">
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  
  <title>项目加载器 &mdash; Scrapy 2.3.0 文档</title>
  

  
  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster.custom.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster.bundle.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-shadow.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-punk.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-noir.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-light.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-borderless.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/micromodal.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/sphinx_rtd_theme.css" type="text/css" />

  
  
  
  

  
  <!--[if lt IE 9]>
    <script src="../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
        <script src="../_static/jquery.js"></script>
        <script src="../_static/underscore.js"></script>
        <script src="../_static/doctools.js"></script>
        <script src="../_static/language_data.js"></script>
        <script src="../_static/js/hoverxref.js"></script>
        <script src="../_static/js/tooltipster.bundle.min.js"></script>
        <script src="../_static/js/micromodal.min.js"></script>
    
    <script type="text/javascript" src="../_static/js/theme.js"></script>

    
    <link rel="index" title="索引" href="../genindex.html" />
    <link rel="search" title="搜索" href="../search.html" />
    <link rel="next" title="Scrapy shell" href="shell.html" />
    <link rel="prev" title="项目" href="items.html" /> 
</head>

<body class="wy-body-for-nav">

   
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >
          

          
            <a href="../index.html" class="icon icon-home" alt="Documentation Home"> Scrapy
          

          
          </a>

          
            
            
              <div class="version">
                2.3
              </div>
            
          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <p class="caption"><span class="caption-text">第一步</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../intro/overview.html">Scrapy一目了然</a></li>
<li class="toctree-l1"><a class="reference internal" href="../intro/install.html">安装指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../intro/tutorial.html">Scrapy 教程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../intro/examples.html">实例</a></li>
</ul>
<p class="caption"><span class="caption-text">基本概念</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="commands.html">命令行工具</a></li>
<li class="toctree-l1"><a class="reference internal" href="spiders.html">蜘蛛</a></li>
<li class="toctree-l1"><a class="reference internal" href="selectors.html">选择器</a></li>
<li class="toctree-l1"><a class="reference internal" href="items.html">项目</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">项目加载器</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#using-item-loaders-to-populate-items">使用项目加载器填充项目</a></li>
<li class="toctree-l2"><a class="reference internal" href="#working-with-dataclass-items">使用dataclass项</a></li>
<li class="toctree-l2"><a class="reference internal" href="#input-and-output-processors">输入和输出处理器</a></li>
<li class="toctree-l2"><a class="reference internal" href="#declaring-item-loaders">声明项加载器</a></li>
<li class="toctree-l2"><a class="reference internal" href="#declaring-input-and-output-processors">声明输入和输出处理器</a></li>
<li class="toctree-l2"><a class="reference internal" href="#item-loader-context">项目加载器上下文</a></li>
<li class="toctree-l2"><a class="reference internal" href="#itemloader-objects">项加载器对象</a></li>
<li class="toctree-l2"><a class="reference internal" href="#nested-loaders">嵌套装载机</a></li>
<li class="toctree-l2"><a class="reference internal" href="#reusing-and-extending-item-loaders">重复使用和扩展项目加载器</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="shell.html">Scrapy shell</a></li>
<li class="toctree-l1"><a class="reference internal" href="item-pipeline.html">项目管道</a></li>
<li class="toctree-l1"><a class="reference internal" href="feed-exports.html">Feed 导出</a></li>
<li class="toctree-l1"><a class="reference internal" href="request-response.html">请求和响应</a></li>
<li class="toctree-l1"><a class="reference internal" href="link-extractors.html">链接提取器</a></li>
<li class="toctree-l1"><a class="reference internal" href="settings.html">设置</a></li>
<li class="toctree-l1"><a class="reference internal" href="exceptions.html">例外情况</a></li>
</ul>
<p class="caption"><span class="caption-text">内置服务</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="logging.html">登录</a></li>
<li class="toctree-l1"><a class="reference internal" href="stats.html">统计数据集合</a></li>
<li class="toctree-l1"><a class="reference internal" href="email.html">发送电子邮件</a></li>
<li class="toctree-l1"><a class="reference internal" href="telnetconsole.html">远程登录控制台</a></li>
<li class="toctree-l1"><a class="reference internal" href="webservice.html">Web服务</a></li>
</ul>
<p class="caption"><span class="caption-text">解决具体问题</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">常见问题</a></li>
<li class="toctree-l1"><a class="reference internal" href="debug.html">调试spiders</a></li>
<li class="toctree-l1"><a class="reference internal" href="contracts.html">蜘蛛合约</a></li>
<li class="toctree-l1"><a class="reference internal" href="practices.html">常用做法</a></li>
<li class="toctree-l1"><a class="reference internal" href="broad-crawls.html">宽爬行</a></li>
<li class="toctree-l1"><a class="reference internal" href="developer-tools.html">使用浏览器的开发人员工具进行抓取</a></li>
<li class="toctree-l1"><a class="reference internal" href="dynamic-content.html">选择动态加载的内容</a></li>
<li class="toctree-l1"><a class="reference internal" href="leaks.html">调试内存泄漏</a></li>
<li class="toctree-l1"><a class="reference internal" href="media-pipeline.html">下载和处理文件和图像</a></li>
<li class="toctree-l1"><a class="reference internal" href="deploy.html">部署蜘蛛</a></li>
<li class="toctree-l1"><a class="reference internal" href="autothrottle.html">AutoThrottle 扩展</a></li>
<li class="toctree-l1"><a class="reference internal" href="benchmarking.html">标杆管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="jobs.html">作业：暂停和恢复爬行</a></li>
<li class="toctree-l1"><a class="reference internal" href="coroutines.html">协同程序</a></li>
<li class="toctree-l1"><a class="reference internal" href="asyncio.html">asyncio</a></li>
</ul>
<p class="caption"><span class="caption-text">扩展Scrapy</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="architecture.html">体系结构概述</a></li>
<li class="toctree-l1"><a class="reference internal" href="downloader-middleware.html">下载器中间件</a></li>
<li class="toctree-l1"><a class="reference internal" href="spider-middleware.html">蜘蛛中间件</a></li>
<li class="toctree-l1"><a class="reference internal" href="extensions.html">扩展</a></li>
<li class="toctree-l1"><a class="reference internal" href="api.html">核心API</a></li>
<li class="toctree-l1"><a class="reference internal" href="signals.html">信号</a></li>
<li class="toctree-l1"><a class="reference internal" href="exporters.html">条目导出器</a></li>
</ul>
<p class="caption"><span class="caption-text">其余所有</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../news.html">发行说明</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">为 Scrapy 贡献</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versioning.html">版本控制和API稳定性</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">Scrapy</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
        
      <li>项目加载器</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
            
        
      </li>
    
  </ul>

  
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
  <div class="section" id="module-scrapy.loader">
<span id="item-loaders"></span><span id="topics-loaders"></span><h1>项目加载器<a class="headerlink" href="#module-scrapy.loader" title="永久链接至标题">¶</a></h1>
<p>物品装载机为填充刮削物提供了一种方便的机制。 <a class="reference internal" href="items.html#topics-items"><span class="std std-ref">items</span></a> . 尽管可以直接填充项，但是条目加载器通过自动执行一些常见任务（如在分配原始提取数据之前对其进行解析），为从抓取过程填充它们提供了一个更为方便的API。</p>
<p>换言之， <a class="reference internal" href="items.html#topics-items"><span class="std std-ref">items</span></a> 提供 <em>容器</em> 当项目加载器为 <em>人口增长</em> 那个容器。</p><script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
<ins class="adsbygoogle"
     style="display:block; text-align:center;"
     data-ad-layout="in-article"
     data-ad-format="fluid"
     data-ad-client="ca-pub-1466963416408457"
     data-ad-slot="8850786025"></ins>
<script>
     (adsbygoogle = window.adsbygoogle || []).push({});
</script>
<p>项目加载器的设计目的是提供一种灵活、高效和简单的机制，通过蜘蛛或源格式（HTML、XML等）扩展和重写不同的字段解析规则，而不会成为维护的噩梦。</p>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>项目加载器是 <a class="reference external" href="https://itemloaders.readthedocs.io/en/latest/">itemloaders</a> 库，通过添加对 <a class="reference internal" href="request-response.html#topics-request-response"><span class="std std-ref">responses</span></a> .</p>
</div>
<div class="section" id="using-item-loaders-to-populate-items">
<h2>使用项目加载器填充项目<a class="headerlink" href="#using-item-loaders-to-populate-items" title="永久链接至标题">¶</a></h2>
<p>要使用项加载器，必须首先实例化它。您可以使用 <a class="reference internal" href="items.html#topics-items"><span class="std std-ref">item object</span></a> 或者没有，在这种情况下 <a class="reference internal" href="items.html#topics-items"><span class="std std-ref">item object</span></a> 在项目加载器中自动创建 <code class="docutils literal notranslate"><span class="pre">__init__</span></code> 方法使用 <a class="reference internal" href="items.html#topics-items"><span class="std std-ref">item</span></a> 中指定的类 <code class="xref py py-attr docutils literal notranslate"><span class="pre">ItemLoader.default_item_class</span></code> 属性。</p>
<p>然后，开始将值收集到项加载器中，通常使用 <a class="reference internal" href="selectors.html#topics-selectors"><span class="std std-ref">Selectors</span></a> . 您可以向同一个项目字段添加多个值；项目加载器稍后将知道如何使用适当的处理函数“联接”这些值。</p>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>收集的数据以列表的形式在内部存储，允许向同一字段添加多个值。如果 <code class="docutils literal notranslate"><span class="pre">item</span></code> 参数是在创建加载程序时传递的，如果项的每个值已经是iterable，则将按原样存储；如果是单个值，则将用列表包装。</p>
</div>
<p>下面是在 <a class="reference internal" href="spiders.html#topics-spiders"><span class="std std-ref">Spider</span></a> ，使用 <a class="reference internal" href="items.html#topics-items-declaring"><span class="std std-ref">Product item</span></a>  <a class="reference internal" href="items.html#topics-items"><span class="std std-ref">Items chapter</span></a> ：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">scrapy.loader</span> <span class="kn">import</span> <span class="n">ItemLoader</span>
<span class="kn">from</span> <span class="nn">myproject.items</span> <span class="kn">import</span> <span class="n">Product</span>

<span class="k">def</span> <span class="nf">parse</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">response</span><span class="p">):</span>
    <span class="n">l</span> <span class="o">=</span> <span class="n">ItemLoader</span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="n">Product</span><span class="p">(),</span> <span class="n">response</span><span class="o">=</span><span class="n">response</span><span class="p">)</span>
    <span class="n">l</span><span class="o">.</span><span class="n">add_xpath</span><span class="p">(</span><span class="s1">&#39;name&#39;</span><span class="p">,</span> <span class="s1">&#39;//div[@class=&quot;product_name&quot;]&#39;</span><span class="p">)</span>
    <span class="n">l</span><span class="o">.</span><span class="n">add_xpath</span><span class="p">(</span><span class="s1">&#39;name&#39;</span><span class="p">,</span> <span class="s1">&#39;//div[@class=&quot;product_title&quot;]&#39;</span><span class="p">)</span>
    <span class="n">l</span><span class="o">.</span><span class="n">add_xpath</span><span class="p">(</span><span class="s1">&#39;price&#39;</span><span class="p">,</span> <span class="s1">&#39;//p[@id=&quot;price&quot;]&#39;</span><span class="p">)</span>
    <span class="n">l</span><span class="o">.</span><span class="n">add_css</span><span class="p">(</span><span class="s1">&#39;stock&#39;</span><span class="p">,</span> <span class="s1">&#39;p#stock]&#39;</span><span class="p">)</span>
    <span class="n">l</span><span class="o">.</span><span class="n">add_value</span><span class="p">(</span><span class="s1">&#39;last_updated&#39;</span><span class="p">,</span> <span class="s1">&#39;today&#39;</span><span class="p">)</span> <span class="c1"># you can also use literal values</span>
    <span class="k">return</span> <span class="n">l</span><span class="o">.</span><span class="n">load_item</span><span class="p">()</span>
</pre></div>
</div>
<p>通过快速查看该代码，我们可以看到 <code class="docutils literal notranslate"><span class="pre">name</span></code> 正在从页面中的两个不同的xpath位置提取字段：</p>
<ol class="arabic simple">
<li><p><code class="docutils literal notranslate"><span class="pre">//div[&#64;class=&quot;product_name&quot;]</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">//div[&#64;class=&quot;product_title&quot;]</span></code></p></li>
</ol>
<p>换句话说，通过使用 <code class="xref py py-meth docutils literal notranslate"><span class="pre">add_xpath()</span></code> 方法。这是将分配给 <code class="docutils literal notranslate"><span class="pre">name</span></code> 以后再说。</p>
<p>之后，类似的呼叫用于 <code class="docutils literal notranslate"><span class="pre">price</span></code> 和 <code class="docutils literal notranslate"><span class="pre">stock</span></code> 字段（后者使用CSS选择器 <code class="xref py py-meth docutils literal notranslate"><span class="pre">add_css()</span></code> 方法），最后 <code class="docutils literal notranslate"><span class="pre">last_update</span></code> 直接用文字值填充字段 (<code class="docutils literal notranslate"><span class="pre">today</span></code> ）使用不同的方法： <code class="xref py py-meth docutils literal notranslate"><span class="pre">add_value()</span></code> .</p>
<p>最后，当收集所有数据时， <code class="xref py py-meth docutils literal notranslate"><span class="pre">ItemLoader.load_item()</span></code> 方法，它实际返回用以前提取和收集的数据填充的项 <code class="xref py py-meth docutils literal notranslate"><span class="pre">add_xpath()</span></code> ， <code class="xref py py-meth docutils literal notranslate"><span class="pre">add_css()</span></code> 和 <code class="xref py py-meth docutils literal notranslate"><span class="pre">add_value()</span></code> 电话。</p>
</div>
<div class="section" id="working-with-dataclass-items">
<span id="topics-loaders-dataclass"></span><h2>使用dataclass项<a class="headerlink" href="#working-with-dataclass-items" title="永久链接至标题">¶</a></h2>
<p>默认情况下， <a class="reference internal" href="items.html#dataclass-items"><span class="std std-ref">dataclass items</span></a> 创建时要求传递所有字段。在使用带有项加载器的dataclass项时，这可能是一个问题：除非将预先填充的项传递给加载程序，否则将使用加载程序的 <code class="xref py py-meth docutils literal notranslate"><span class="pre">add_xpath()</span></code> ， <code class="xref py py-meth docutils literal notranslate"><span class="pre">add_css()</span></code> 和 <code class="xref py py-meth docutils literal notranslate"><span class="pre">add_value()</span></code> 方法。</p>
<p>克服这个问题的一种方法是使用 <a class="reference external" href="https://docs.python.org/3/library/dataclasses.html#dataclasses.field" title="(在 Python v3.9)"><code class="xref py py-func docutils literal notranslate"><span class="pre">field()</span></code></a> 函数，带有 <code class="docutils literal notranslate"><span class="pre">default</span></code> 论点：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">field</span>
<span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Optional</span>

<span class="nd">@dataclass</span>
<span class="k">class</span> <span class="nc">InventoryItem</span><span class="p">:</span>
    <span class="n">name</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
    <span class="n">price</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">float</span><span class="p">]</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
    <span class="n">stock</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">int</span><span class="p">]</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="input-and-output-processors">
<span id="topics-loaders-processors"></span><h2>输入和输出处理器<a class="headerlink" href="#input-and-output-processors" title="永久链接至标题">¶</a></h2>
<p>项目加载器为每个（项目）字段包含一个输入处理器和一个输出处理器。输入处理器一旦接收到提取的数据（通过 <code class="xref py py-meth docutils literal notranslate"><span class="pre">add_xpath()</span></code> ， <code class="xref py py-meth docutils literal notranslate"><span class="pre">add_css()</span></code> 或 <code class="xref py py-meth docutils literal notranslate"><span class="pre">add_value()</span></code> 方法），输入处理器的结果被收集并保存在itemloader中。在收集所有数据之后， <code class="xref py py-meth docutils literal notranslate"><span class="pre">ItemLoader.load_item()</span></code> 方法来填充和获取填充的 <a class="reference internal" href="items.html#topics-items"><span class="std std-ref">item object</span></a> .  此时，将使用先前收集的数据（并使用输入处理器处理）调用输出处理器。输出处理器的结果是分配给项目的最终值。</p>
<p>让我们看一个例子来说明如何为一个特定的字段调用输入和输出处理器（这同样适用于任何其他字段）：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">l</span> <span class="o">=</span> <span class="n">ItemLoader</span><span class="p">(</span><span class="n">Product</span><span class="p">(),</span> <span class="n">some_selector</span><span class="p">)</span>
<span class="n">l</span><span class="o">.</span><span class="n">add_xpath</span><span class="p">(</span><span class="s1">&#39;name&#39;</span><span class="p">,</span> <span class="n">xpath1</span><span class="p">)</span> <span class="c1"># (1)</span>
<span class="n">l</span><span class="o">.</span><span class="n">add_xpath</span><span class="p">(</span><span class="s1">&#39;name&#39;</span><span class="p">,</span> <span class="n">xpath2</span><span class="p">)</span> <span class="c1"># (2)</span>
<span class="n">l</span><span class="o">.</span><span class="n">add_css</span><span class="p">(</span><span class="s1">&#39;name&#39;</span><span class="p">,</span> <span class="n">css</span><span class="p">)</span> <span class="c1"># (3)</span>
<span class="n">l</span><span class="o">.</span><span class="n">add_value</span><span class="p">(</span><span class="s1">&#39;name&#39;</span><span class="p">,</span> <span class="s1">&#39;test&#39;</span><span class="p">)</span> <span class="c1"># (4)</span>
<span class="k">return</span> <span class="n">l</span><span class="o">.</span><span class="n">load_item</span><span class="p">()</span> <span class="c1"># (5)</span>
</pre></div>
</div>
<p>所以发生的是：</p>
<ol class="arabic simple">
<li><p>数据来自 <code class="docutils literal notranslate"><span class="pre">xpath1</span></code> 提取并通过 <em>输入处理器</em> 的 <code class="docutils literal notranslate"><span class="pre">name</span></code> 字段。输入处理器的结果被收集并保存在项目加载器中（但尚未分配给项目）。</p></li>
<li><p>数据来自 <code class="docutils literal notranslate"><span class="pre">xpath2</span></code> 提取并通过 <em>输入处理器</em> 用于（1）。输入处理器的结果将附加到（1）中收集的数据（如果有）中。</p></li>
<li><p>这种情况与以前的情况类似，只是数据是从 <code class="docutils literal notranslate"><span class="pre">css</span></code> 并通过相同的 <em>输入处理器</em> 用于（1）和（2）。输入处理器的结果将附加到（1）和（2）中收集的数据（如果有）中。</p></li>
<li><p>这种情况也与前面的情况类似，只是要收集的值是直接分配的，而不是从xpath表达式或css选择器中提取的。但是，该值仍然通过输入处理器传递。在这种情况下，由于该值不可重设，因此在将其传递给输入处理器之前，它将转换为单个元素的可重设值，因为输入处理器始终接收可重设值。</p></li>
<li><p>步骤（1）、（2）、（3）和（4）中收集的数据通过 <em>输出处理器</em> 的 <code class="docutils literal notranslate"><span class="pre">name</span></code> 字段。输出处理器的结果是分配给 <code class="docutils literal notranslate"><span class="pre">name</span></code> 项目中的字段。</p></li>
</ol>
<p>值得注意的是，处理器只是可调用的对象，它们与要解析的数据一起调用，并返回已解析的值。所以你可以使用任何函数作为输入或输出处理器。唯一的要求是它们必须接受一个（而且只有一个）位置参数，这将是一个iterable。</p>
<div class="versionchanged">
<p><span class="versionmodified changed">在 2.0 版更改: </span>处理器不再需要是方法。</p>
</div>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>输入和输出处理器都必须接收iterable作为其第一个参数。这些函数的输出可以是任何东西。输入处理器的结果将被附加到一个内部列表中（在加载程序中），其中包含收集的值（对于该字段）。输出处理器的结果是最终分配给项目的值。</p>
</div>
<p>您需要记住的另一件事是，输入处理器返回的值在内部收集（在列表中），然后传递给输出处理器来填充字段。</p>
<p>最后，但并非最不重要， <a class="reference external" href="https://itemloaders.readthedocs.io/en/latest/">itemloaders</a> 有一些 <a class="reference external" href="https://itemloaders.readthedocs.io/en/latest/built-in-processors.html#built-in-processors" title="(在 itemloaders)"><span class="xref std std-ref">commonly used processors</span></a> 内置方便。</p>
</div>
<div class="section" id="declaring-item-loaders">
<h2>声明项加载器<a class="headerlink" href="#declaring-item-loaders" title="永久链接至标题">¶</a></h2>
<p>项加载器使用类定义语法声明。举个例子：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">itemloaders.processors</span> <span class="kn">import</span> <span class="n">TakeFirst</span><span class="p">,</span> <span class="n">MapCompose</span><span class="p">,</span> <span class="n">Join</span>
<span class="kn">from</span> <span class="nn">scrapy.loader</span> <span class="kn">import</span> <span class="n">ItemLoader</span>

<span class="k">class</span> <span class="nc">ProductLoader</span><span class="p">(</span><span class="n">ItemLoader</span><span class="p">):</span>

    <span class="n">default_output_processor</span> <span class="o">=</span> <span class="n">TakeFirst</span><span class="p">()</span>

    <span class="n">name_in</span> <span class="o">=</span> <span class="n">MapCompose</span><span class="p">(</span><span class="nb">str</span><span class="o">.</span><span class="n">title</span><span class="p">)</span>
    <span class="n">name_out</span> <span class="o">=</span> <span class="n">Join</span><span class="p">()</span>

    <span class="n">price_in</span> <span class="o">=</span> <span class="n">MapCompose</span><span class="p">(</span><span class="nb">str</span><span class="o">.</span><span class="n">strip</span><span class="p">)</span>

    <span class="c1"># ...</span>
</pre></div>
</div>
<p>如您所见，输入处理器使用 <code class="docutils literal notranslate"><span class="pre">_in</span></code> 当输出处理器使用 <code class="docutils literal notranslate"><span class="pre">_out</span></code> 后缀。您还可以使用 <code class="xref py py-attr docutils literal notranslate"><span class="pre">ItemLoader.default_input_processor</span></code> 和 <code class="xref py py-attr docutils literal notranslate"><span class="pre">ItemLoader.default_output_processor</span></code> 属性。</p>
</div>
<div class="section" id="declaring-input-and-output-processors">
<span id="topics-loaders-processors-declaring"></span><h2>声明输入和输出处理器<a class="headerlink" href="#declaring-input-and-output-processors" title="永久链接至标题">¶</a></h2>
<p>如前一节所述，可以在项目加载器定义中声明输入和输出处理器，并且用这种方式声明输入处理器是非常常见的。但是，还有一个地方可以指定要使用的输入和输出处理器：在 <a class="reference internal" href="items.html#topics-items-fields"><span class="std std-ref">Item Field</span></a> 元数据。下面是一个例子：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">scrapy</span>
<span class="kn">from</span> <span class="nn">itemloaders.processors</span> <span class="kn">import</span> <span class="n">Join</span><span class="p">,</span> <span class="n">MapCompose</span><span class="p">,</span> <span class="n">TakeFirst</span>
<span class="kn">from</span> <span class="nn">w3lib.html</span> <span class="kn">import</span> <span class="n">remove_tags</span>

<span class="k">def</span> <span class="nf">filter_price</span><span class="p">(</span><span class="n">value</span><span class="p">):</span>
    <span class="k">if</span> <span class="n">value</span><span class="o">.</span><span class="n">isdigit</span><span class="p">():</span>
        <span class="k">return</span> <span class="n">value</span>

<span class="k">class</span> <span class="nc">Product</span><span class="p">(</span><span class="n">scrapy</span><span class="o">.</span><span class="n">Item</span><span class="p">):</span>
    <span class="n">name</span> <span class="o">=</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Field</span><span class="p">(</span>
        <span class="n">input_processor</span><span class="o">=</span><span class="n">MapCompose</span><span class="p">(</span><span class="n">remove_tags</span><span class="p">),</span>
        <span class="n">output_processor</span><span class="o">=</span><span class="n">Join</span><span class="p">(),</span>
    <span class="p">)</span>
    <span class="n">price</span> <span class="o">=</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Field</span><span class="p">(</span>
        <span class="n">input_processor</span><span class="o">=</span><span class="n">MapCompose</span><span class="p">(</span><span class="n">remove_tags</span><span class="p">,</span> <span class="n">filter_price</span><span class="p">),</span>
        <span class="n">output_processor</span><span class="o">=</span><span class="n">TakeFirst</span><span class="p">(),</span>
    <span class="p">)</span>
</pre></div>
</div>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">scrapy.loader</span> <span class="kn">import</span> <span class="n">ItemLoader</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">il</span> <span class="o">=</span> <span class="n">ItemLoader</span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="n">Product</span><span class="p">())</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">il</span><span class="o">.</span><span class="n">add_value</span><span class="p">(</span><span class="s1">&#39;name&#39;</span><span class="p">,</span> <span class="p">[</span><span class="s1">&#39;Welcome to my&#39;</span><span class="p">,</span> <span class="s1">&#39;&lt;strong&gt;website&lt;/strong&gt;&#39;</span><span class="p">])</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">il</span><span class="o">.</span><span class="n">add_value</span><span class="p">(</span><span class="s1">&#39;price&#39;</span><span class="p">,</span> <span class="p">[</span><span class="s1">&#39;&amp;euro;&#39;</span><span class="p">,</span> <span class="s1">&#39;&lt;span&gt;1000&lt;/span&gt;&#39;</span><span class="p">])</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">il</span><span class="o">.</span><span class="n">load_item</span><span class="p">()</span>
<span class="go">{&#39;name&#39;: &#39;Welcome to my website&#39;, &#39;price&#39;: &#39;1000&#39;}</span>
</pre></div>
</div>
<p>输入和输出处理器的优先顺序如下：</p>
<ol class="arabic simple">
<li><p>项目加载器字段特定属性： <code class="docutils literal notranslate"><span class="pre">field_in</span></code> 和 <code class="docutils literal notranslate"><span class="pre">field_out</span></code> （最优先）</p></li>
<li><p>字段元数据 (<code class="docutils literal notranslate"><span class="pre">input_processor</span></code> 和 <code class="docutils literal notranslate"><span class="pre">output_processor</span></code> 关键）</p></li>
<li><p>项目加载器默认值： <code class="xref py py-meth docutils literal notranslate"><span class="pre">ItemLoader.default_input_processor()</span></code> 和 <code class="xref py py-meth docutils literal notranslate"><span class="pre">ItemLoader.default_output_processor()</span></code> （最低优先级）</p></li>
</ol>
<p>参见： <a class="reference internal" href="#topics-loaders-extending"><span class="std std-ref">重复使用和扩展项目加载器</span></a> .</p>
</div>
<div class="section" id="item-loader-context">
<span id="topics-loaders-context"></span><h2>项目加载器上下文<a class="headerlink" href="#item-loader-context" title="永久链接至标题">¶</a></h2>
<p>项目加载器上下文是任意键/值的dict，在项目加载器中的所有输入和输出处理器之间共享。它可以在声明、实例化或使用项加载器时传递。它们用于修改输入/输出处理器的行为。</p>
<p>例如，假设您有一个函数 <code class="docutils literal notranslate"><span class="pre">parse_length</span></code> 它接收文本值并从中提取长度：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">parse_length</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">loader_context</span><span class="p">):</span>
    <span class="n">unit</span> <span class="o">=</span> <span class="n">loader_context</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;unit&#39;</span><span class="p">,</span> <span class="s1">&#39;m&#39;</span><span class="p">)</span>
    <span class="c1"># ... length parsing code goes here ...</span>
    <span class="k">return</span> <span class="n">parsed_length</span>
</pre></div>
</div>
<p>接受一个 <code class="docutils literal notranslate"><span class="pre">loader_context</span></code> 参数函数显式地告诉项目加载器它能够接收项目加载器上下文，因此项目加载器在调用它时传递当前活动的上下文，以及处理器函数 (<code class="docutils literal notranslate"><span class="pre">parse_length</span></code> 在这种情况下）可以使用它们。</p>
<p>修改项目加载器上下文值有几种方法：</p>
<ol class="arabic">
<li><p>通过修改当前活动的项加载器上下文 (<code class="xref py py-attr docutils literal notranslate"><span class="pre">context</span></code> 属性）：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">loader</span> <span class="o">=</span> <span class="n">ItemLoader</span><span class="p">(</span><span class="n">product</span><span class="p">)</span>
<span class="n">loader</span><span class="o">.</span><span class="n">context</span><span class="p">[</span><span class="s1">&#39;unit&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="s1">&#39;cm&#39;</span>
</pre></div>
</div>
</li>
<li><p>项目加载器实例化（项目加载器的关键字参数 <code class="docutils literal notranslate"><span class="pre">__init__</span></code> 方法存储在项加载器上下文中）：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">loader</span> <span class="o">=</span> <span class="n">ItemLoader</span><span class="p">(</span><span class="n">product</span><span class="p">,</span> <span class="n">unit</span><span class="o">=</span><span class="s1">&#39;cm&#39;</span><span class="p">)</span>
</pre></div>
</div>
</li>
<li><p>在项目加载器声明中，用于那些支持用项目加载器上下文实例化它们的输入/输出处理器。 <code class="xref py py-class docutils literal notranslate"><span class="pre">MapCompose</span></code> 是其中之一：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">ProductLoader</span><span class="p">(</span><span class="n">ItemLoader</span><span class="p">):</span>
    <span class="n">length_out</span> <span class="o">=</span> <span class="n">MapCompose</span><span class="p">(</span><span class="n">parse_length</span><span class="p">,</span> <span class="n">unit</span><span class="o">=</span><span class="s1">&#39;cm&#39;</span><span class="p">)</span>
</pre></div>
</div>
</li>
</ol>
</div>
<div class="section" id="itemloader-objects">
<h2>项加载器对象<a class="headerlink" href="#itemloader-objects" title="永久链接至标题">¶</a></h2>
</div>
<div class="section" id="nested-loaders">
<span id="topics-loaders-nested"></span><h2>嵌套装载机<a class="headerlink" href="#nested-loaders" title="永久链接至标题">¶</a></h2>
<p>从文档的子部分分析相关值时，创建嵌套加载器可能很有用。假设您正在从一个页面的页脚提取细节，该页面的外观如下：</p>
<p>例子：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">&lt;</span><span class="n">footer</span><span class="o">&gt;</span>
    <span class="o">&lt;</span><span class="n">a</span> <span class="n">class</span><span class="o">=</span><span class="s2">&quot;social&quot;</span> <span class="n">href</span><span class="o">=</span><span class="s2">&quot;https://facebook.com/whatever&quot;</span><span class="o">&gt;</span><span class="n">Like</span> <span class="n">Us</span><span class="o">&lt;/</span><span class="n">a</span><span class="o">&gt;</span>
    <span class="o">&lt;</span><span class="n">a</span> <span class="n">class</span><span class="o">=</span><span class="s2">&quot;social&quot;</span> <span class="n">href</span><span class="o">=</span><span class="s2">&quot;https://twitter.com/whatever&quot;</span><span class="o">&gt;</span><span class="n">Follow</span> <span class="n">Us</span><span class="o">&lt;/</span><span class="n">a</span><span class="o">&gt;</span>
    <span class="o">&lt;</span><span class="n">a</span> <span class="n">class</span><span class="o">=</span><span class="s2">&quot;email&quot;</span> <span class="n">href</span><span class="o">=</span><span class="s2">&quot;mailto:whatever@example.com&quot;</span><span class="o">&gt;</span><span class="n">Email</span> <span class="n">Us</span><span class="o">&lt;/</span><span class="n">a</span><span class="o">&gt;</span>
<span class="o">&lt;/</span><span class="n">footer</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>如果没有嵌套加载程序，则需要为要提取的每个值指定完整的xpath（或css）。</p>
<p>例子：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">loader</span> <span class="o">=</span> <span class="n">ItemLoader</span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="n">Item</span><span class="p">())</span>
<span class="c1"># load stuff not in the footer</span>
<span class="n">loader</span><span class="o">.</span><span class="n">add_xpath</span><span class="p">(</span><span class="s1">&#39;social&#39;</span><span class="p">,</span> <span class="s1">&#39;//footer/a[@class = &quot;social&quot;]/@href&#39;</span><span class="p">)</span>
<span class="n">loader</span><span class="o">.</span><span class="n">add_xpath</span><span class="p">(</span><span class="s1">&#39;email&#39;</span><span class="p">,</span> <span class="s1">&#39;//footer/a[@class = &quot;email&quot;]/@href&#39;</span><span class="p">)</span>
<span class="n">loader</span><span class="o">.</span><span class="n">load_item</span><span class="p">()</span>
</pre></div>
</div>
<p>相反，您可以使用页脚选择器创建嵌套加载程序，并添加相对于页脚的值。功能相同，但避免重复页脚选择器。</p>
<p>例子：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">loader</span> <span class="o">=</span> <span class="n">ItemLoader</span><span class="p">(</span><span class="n">item</span><span class="o">=</span><span class="n">Item</span><span class="p">())</span>
<span class="c1"># load stuff not in the footer</span>
<span class="n">footer_loader</span> <span class="o">=</span> <span class="n">loader</span><span class="o">.</span><span class="n">nested_xpath</span><span class="p">(</span><span class="s1">&#39;//footer&#39;</span><span class="p">)</span>
<span class="n">footer_loader</span><span class="o">.</span><span class="n">add_xpath</span><span class="p">(</span><span class="s1">&#39;social&#39;</span><span class="p">,</span> <span class="s1">&#39;a[@class = &quot;social&quot;]/@href&#39;</span><span class="p">)</span>
<span class="n">footer_loader</span><span class="o">.</span><span class="n">add_xpath</span><span class="p">(</span><span class="s1">&#39;email&#39;</span><span class="p">,</span> <span class="s1">&#39;a[@class = &quot;email&quot;]/@href&#39;</span><span class="p">)</span>
<span class="c1"># no need to call footer_loader.load_item()</span>
<span class="n">loader</span><span class="o">.</span><span class="n">load_item</span><span class="p">()</span>
</pre></div>
</div>
<p>您可以任意嵌套加载程序，它们可以使用xpath或css选择器。作为一般准则，当嵌套加载器使您的代码更简单，但不要过度嵌套，否则您的解析器可能会变得难以读取。</p>
</div>
<div class="section" id="reusing-and-extending-item-loaders">
<span id="topics-loaders-extending"></span><h2>重复使用和扩展项目加载器<a class="headerlink" href="#reusing-and-extending-item-loaders" title="永久链接至标题">¶</a></h2>
<p>随着项目规模的扩大和蜘蛛数量的增加，维护成为一个基本问题，特别是当您必须为每个蜘蛛处理许多不同的解析规则时，有许多异常，但也希望重用公共处理器。</p>
<p>项目加载器旨在减轻解析规则的维护负担，而不会失去灵活性，同时为扩展和重写规则提供了方便的机制。因此，项目加载器支持传统的Python类继承来处理特定spider（或spider组）的差异。</p>
<p>例如，假设某个特定站点用三个破折号（例如 <code class="docutils literal notranslate"><span class="pre">---Plasma</span> <span class="pre">TV---</span></code> ）你不想最后在最终的产品名称中删除这些破折号。</p>
<p>下面介绍如何通过重用和扩展默认的产品项加载器来删除这些破折号。 (<code class="docutils literal notranslate"><span class="pre">ProductLoader</span></code> ）：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">itemloaders.processors</span> <span class="kn">import</span> <span class="n">MapCompose</span>
<span class="kn">from</span> <span class="nn">myproject.ItemLoaders</span> <span class="kn">import</span> <span class="n">ProductLoader</span>

<span class="k">def</span> <span class="nf">strip_dashes</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
    <span class="k">return</span> <span class="n">x</span><span class="o">.</span><span class="n">strip</span><span class="p">(</span><span class="s1">&#39;-&#39;</span><span class="p">)</span>

<span class="k">class</span> <span class="nc">SiteSpecificLoader</span><span class="p">(</span><span class="n">ProductLoader</span><span class="p">):</span>
    <span class="n">name_in</span> <span class="o">=</span> <span class="n">MapCompose</span><span class="p">(</span><span class="n">strip_dashes</span><span class="p">,</span> <span class="n">ProductLoader</span><span class="o">.</span><span class="n">name_in</span><span class="p">)</span>
</pre></div>
</div>
<p>扩展项加载器非常有用的另一种情况是当您有多个源格式时，例如XML和HTML。在XML版本中，您可能希望删除 <code class="docutils literal notranslate"><span class="pre">CDATA</span></code> 发生。下面是一个如何操作的示例：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">itemloaders.processors</span> <span class="kn">import</span> <span class="n">MapCompose</span>
<span class="kn">from</span> <span class="nn">myproject.ItemLoaders</span> <span class="kn">import</span> <span class="n">ProductLoader</span>
<span class="kn">from</span> <span class="nn">myproject.utils.xml</span> <span class="kn">import</span> <span class="n">remove_cdata</span>

<span class="k">class</span> <span class="nc">XmlProductLoader</span><span class="p">(</span><span class="n">ProductLoader</span><span class="p">):</span>
    <span class="n">name_in</span> <span class="o">=</span> <span class="n">MapCompose</span><span class="p">(</span><span class="n">remove_cdata</span><span class="p">,</span> <span class="n">ProductLoader</span><span class="o">.</span><span class="n">name_in</span><span class="p">)</span>
</pre></div>
</div>
<p>这就是您通常如何扩展输入处理器的方法。</p>
<p>对于输出处理器，在字段元数据中声明它们更为常见，因为它们通常只依赖于字段，而不依赖于每个特定的站点解析规则（与输入处理器一样）。参见： <a class="reference internal" href="#topics-loaders-processors-declaring"><span class="std std-ref">声明输入和输出处理器</span></a> .</p>
<p>有许多其他可能的方法来扩展、继承和重写项加载器，不同的项加载器层次结构可能更适合不同的项目。Scrapy只提供了这种机制；它不会强制任何特定的装载机集合组织——这取决于您和项目的需要。</p>
</div>
</div>


           </div>
           
          </div>
          <footer>
  
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
      
        <a href="shell.html" class="btn btn-neutral float-right" title="Scrapy shell" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
      
      
        <a href="items.html" class="btn btn-neutral float-left" title="项目" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
      
    </div>
  

  <hr/>

  <div role="contentinfo">
    <p>
        
        &copy; 版权所有 2008–2020, Scrapy developers
      <span class="lastupdated">
        最后更新于 10月 18, 2020.
      </span>

    </p>
  </div>
    
    
    
    Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a
    
    <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a>
    
    provided by <a href="https://readthedocs.org">Read the Docs</a>. 

</footer>

        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
  
 
<script type="text/javascript">
!function(){var analytics=window.analytics=window.analytics||[];if(!analytics.initialize)if(analytics.invoked)window.console&&console.error&&console.error("Segment snippet included twice.");else{analytics.invoked=!0;analytics.methods=["trackSubmit","trackClick","trackLink","trackForm","pageview","identify","reset","group","track","ready","alias","page","once","off","on"];analytics.factory=function(t){return function(){var e=Array.prototype.slice.call(arguments);e.unshift(t);analytics.push(e);return analytics}};for(var t=0;t<analytics.methods.length;t++){var e=analytics.methods[t];analytics[e]=analytics.factory(e)}analytics.load=function(t){var e=document.createElement("script");e.type="text/javascript";e.async=!0;e.src=("https:"===document.location.protocol?"https://":"http://")+"cdn.segment.com/analytics.js/v1/"+t+"/analytics.min.js";var n=document.getElementsByTagName("script")[0];n.parentNode.insertBefore(e,n)};analytics.SNIPPET_VERSION="3.1.0";
analytics.load("8UDQfnf3cyFSTsM4YANnW5sXmgZVILbA");
analytics.page();
}}();

analytics.ready(function () {
    ga('require', 'linker');
    ga('linker:autoLink', ['scrapinghub.com', 'crawlera.com']);
});
</script>


</body>
</html>